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integrated magnetic fields 

Abstract -The transversal propagation of the edge states in a two-dimensional quantum spin 
Hall (QSH) system is classified by the characteristic parameter A. There are two different types 
of the helical edge states, the normal and special edge states, exhibiting distinct behaviors. The 
penetration depth of the normal edge state is momentum dependent, and the finite gap for edge 
band decays monotonously with sample width, leading to the normal finite size effect. In contrast, 
the penetration depth maintains a uniform minimal value in the special edge states, and conse- 
quently the finite gap decays non-monotonously with sample width, leading to the anomalous 
finite size effect. To demonstrate their difference explicitly, we compared the real materials in 
phase diagram. An intuitive way to search for the special edge states in the two-dimensional QSH 
system is also proposed. 
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Introduction. — Owing to the linear dispersion and 
Zi topological invariant, the anomalous transport prop- 
erties of the helical edge states in a quantum spin Hall 
(QSH)system are predicted [1-6]. Well localized edge 
states were usually treated as ideal one-dimensional chan- 
nels to investigate the exotic properties [2, 5]. How- 
ever, their behaviors are significantly changed due to the 
transversal broadening of these edge states in real samples. 
The penetration depth of helical edge states had been dis- 
cussed in both HgTe quantum well system and Bi thin 
film [7]. In those two systems, its value are determined 
by the inverse of momentum space distance between the 
edge state and the absorption point into the bulk. The 
finite penetration depth also leads to the so-called finite 
size effect in two-dimensional (2D) QSH system [8]. A gap 
opens at T point when the opposite helical edge channels 
overlap each other, which had been used to confirm the 
intrinsic spin Hall effect in HgTe quantum well system [9]. 
Recently, several electric devices had been designed to ma- 
nipulate the charge and spin transport with such finite size 
effect [10, 11]. An anomalous finite size effect was further 
reported in three-dimensional (3D) topological insulator 



Bi2Se3, much shorter penetration depth and oscillatory 
finite size gap had been revealed [12]. This gap oscillation 
had also been used to search new candidate of topologi- 
cal non-trivial systems [13,14]. These previous discussions 
were all based on the specific materials. The general com- 
prehension of the transversal propagation behaviors is ex- 
pected in QSH system, especially the intrinsic difference 
between the normal and anomalous finite size effects. 

In this paper, the transversal propagation behaviors of 
the helical edge states in 2D QSH system are investigated. 
The helical edge states can be classified into two modes, 
the normal and special edge states, according to the decay 
characteristic quantity A. In normal edge states, the pen- 
etration depth shows clear momentum-dependence, and 
the finite gap for edge states decays monotonously with 
the sample width. While in special edge states, the pen- 
etration depth keeps unchanged in the momentum space, 
and its finite gap decays oscilatorily with sample width. 
The normal and anomalous finite size effect can be found 
in the respective edge states. These facts give explicit ex- 
planations on the difference between the real materials. 
Based on the theoretical calculations, the search of the 
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special edge states in the 2D case is proposed. 

The paper is organized as follows: In Sec. II, we spec- 
ify two different transversal propagation modes of the he- 
lical edge states without specific boundary condition. A 
scmi-finitc boundary condition is adopted to show the dis- 
tinct evolution of the penetration depth in the normal and 
special edge states in Sec. III. As a consequent effect, the 
normal and anomalous finite size effects are discussed in 
Sec. IV. In Sec.V, the role of particle-hole asymmetry and 
the comparison of real materials are further discussed. 
The conclusion is drawn in Sec. VI. 

Transversal Modes of 2D QSH Model. The 

QSH effect was theoretically predicted in HgTe quantum 
well [3,15], and soon confirmed experimentally by Konig 
el at. [16]. We start from the effective 4x4 model for a 2D 
QSH system proposed by Bernevig, Hughes, and Zhang 
[3,5]. Very recently, it was also adopted as an effective 2D 
model for the 3D topological insulator in ultrathin limit 
[17]. The model Hamiltonian is expressed as 



Table 1: Combination for Ai,a 



H(k) 



h(k) 






h*(-k) 



where h(k) = £fcI 2X 2 + d fe • er, with e k = C - D(k 2 
The k-dependent effective field dj, = (Ak x ,—Ak 



u ■■ 



(1) 



M k ), 



where Mk = M — B{k 2 + k 2 ,). <x is the Pauli matrices. 
A, B, C, D, and M are determined by the quantum well 
geometry in real materials. Here, we treat them as inde- 
pendent parameters and study their respective role first. 
Keep in mind that, the interested topological non-trivial 
QSH phase [18] emerges only when MB > 0. In HgTe 
quantum well system, such condition is controlled by the 
thickness of the quantum well [3, 16]. The properties of 
counter-part h*(—k) can be conveniently obtained by ap- 
plying the time reversal operation to hk- 

To focus on the edge properties, k y needs to be replaced 
by —id y , while k x remains a good quantum number due to 
the translational symmetry. A trial solution of $k x (y) = 
Cfcx e_Ay can be introduced, and the decay characteristic 
quantities are subsequently obtained as [8] 



\U{k x ,E) = ki + F{E)± x F*{E) 



E 2 - M 2 
B 2 - D 2 



(2) 



where F(E) = — ^r^d'*)^ is a function of energy E. 

The transversal propagation behaviors of the states are 
determined by both of Ai.2- However, it is clear in Eqs. (2) 
that, \{k x ,E) has a definite distribution in momentum 
space, independent of the boundary condition. Hence, we 
can directly discuss the transversal propagation behaviors 
of the states from Eqs. (2). There are four modes of the 
states specified by different combinations of Ai ;2 , as illus- 
trated in TABLE 1. 



Mode* 


Condition 


Ai 


A 2 


Edgel 


\\ > X 2 2 > 


Real 


Real 


Bulkl" 


X\ > > \\ 


Real 


Imaginary 


Bulk2 


> \\ > \\ 


Imaginary 


Imaginary 


Edge2 


\\ = (AD* 


Complex 


Complex 



For bulk states, at least one of A12 is purely imaginary. While 
for edge states, both of A1.2 should have a non-zero real part. 
** Trivial edge states are also included due to the real Ai. 

Since only the non-trivial edge states are interested in 
this paper, it is natural to ask whether there is something 
different between the two edge modes in TABLE 1. For 
convenience, we specify the Edgel state with both Ai j2 real 
as the normal edge state (NES); and the Edge2 state with 
A1.2 complex conjugates as the special edge state (SES). 
To fulfill the condition of conjugate, the term under square 
root in Eqs. (2) must be negative, which restricts SES 
existing in a specific regime in momentum space confined 
by 



E S ± ES = D 



A 2 



2B 



"».) ± |^SBF 



A?) (3) 



Here a screen factor denoting the particle-hole asymmetry 
7 = 1 — -g2 is introduced. The system undergoes a phase 
transition from an insulator to a semimetal when D > B 
[19], which is not interested for us. Eqs. (3) naturally re- 
quires 4MB > A 2 > 0, implying a non-trivial QSH state. 
There is no special restriction for the NES. 

Present classification is a natrual consequence of the 
breaking of periodic boundary condition, which leads to 
a definite distribution of transversal propagation modes 
in momentum space. An explicit boundary condition just 
creates a specific spectrum onto such distribution. The 
emergence of SES is determined by the fc^-independent 
jtjSES _ p Qr gj ven parameters, E± ES will squish the bulk 
band, leading to a flat valence band top (or conduct band 
bottom), as in Fig. l(c)(e)(f). Conversely, such feature can 
be viewed as a sufficient condition for SES in QSH system, 
even without the explicit knowledge of material parame- 
ters. Moreover, the squished bulk band also gives rise to 
the Bulk2 states in TABLE 1, which exhibit a larger den- 
sity of states than Bulkl states in our numerical results. 
Such classification is also significant in the problems of in- 
terference tunneling and restricted edge transport [11,20]. 
More differences of the edge modes will be discussed in 
the following sections. 

Penetration Depth of Helical Edge States. 

The penetration depth distribution of the helical edge 
states is distinct from that of the chiral edge states in inte- 
ger quantum Hall effect [21,22]. The latter is determined 
by the universal magnetic length, which is related to the 
external magnetic field. In contrast, the penetration depth 
in QSH system is /c-dependent, originated from the band 
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Fig. 1: Helical edge spectra for different effective parameters. A = 0.4 and B — —1.0. The upper/lower panels are spectra 
with/without particle-hole symmetry. \M\ increases from left to right. The upper, and lower part in each panel is the edge 
spectrum, and corresponding Ai,2 respectively. The bold solid line in black/blue represents the edge spectrum of NES/SES. 
The lighter/darker gray regimes describe the Bulkf/Bulk2 states in Table 1. The light grey dash line are the confines of SES 
given by Eqs. (3). The red solid lines in lower part of each panel is which give the inverse of I. The blue dash lines are 

the corresponding SJAi^- In panel (c), three states are specified, and corresponding transversal propagation behaviors in real 
space are shown in Fig. 2. 



structure [5]. To address this, the semi- infinite boundary 
condition [7] is adopted here. We restrict 9?Ai,2 > to 
obtain an evanescent edge state localized near the bound- 
ary. The boundary condition ^/^(O) = gives the linear 
dispersion relation [7, 8] 

E = -A^jk x - ^ (4) 

The penetration depth i = max{^.\^\} [6] behaves dif- 
ferently in NES and SES. The NES situation had been 
discussed in previous work as in Fig. 1(a) and (d). The /in- 
dependent £nes reaches its minimum at k x = ^-(1— 7 _1 ). 
The edge state is absorbed by the bulk when 5RA2 = 0. In 
contrast, the penetration depth in SES maintains a uni- 
form minimal value across the whole regime of SES, given 

by 

isES = 2/(Ai + A 2 ) = 1 2 -^- 1, (5) 

which is also independent of E, k x and M, as shown in 
Fig. 1. Here 5RAi.2 governs the transversal decay behav- 



ior. In fact, although the relation Ai + A2 = -^=g keeps 
unchanged even in NES, the penetration depth in NES is 
merely determined by the minimum of real Ai^- 

In Fig. 2, the transversal propagation behaviors of three 
selected helical edge states are plotted in real space. The 
wave function of SES(l) and SES(2) exhibit an evanescent 
oscillation with different periods. However, they share the 
same penetration depth. In contrast, the NES (3) shows no 
oscillation but much longer penetration depth. The non- 
monotonous decay behavior of edge state was also reported 
in the lattice model [23] , which can be naturally attributed 
to the SES. 

The HgTe quantum well [3] and the ultra-thin BiiSe?, 
film [17] correspond to the situation in Fig. 1(d), where 
the SES is absent. The penetration depth is estimated to 
be about 50 nm [8]. In previous studies, [7] the Bi{lll} 
thin film was compared with the HgTe quantum well sys- 
tem, and remarkable difference was found in the behaviors 
of the penetration depth. We notice that, a flat valence 
band top emerges in Bi{lll} spectrum [7], implying the 
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Fig. 2: Transversal propagation behaviors of edge state wave 
function in real space. Corresponding states are marked in 
Fig. 1(c). The dash line roughly gives the penetration depth 
behaviors with function exp(—y/£). The black dash line takes 
£ses given by Eqs. (5), while the blue one takes I — SRA^ 1 . 



existence of SES. Therefore, such difference can be well 
understood within present discussion. Due to the similar- 
ity of A at T point, the topological surface states (TSS) of 
the 3D topological insulator can be equivalently discussed 
within our framework, corresponding to the situation in 
Fig. 1(f), where the edge states are SES dominated. The 
penetration depth of TSS in 3D BiiSe-$ was also reported 
in previous work, with a shorter i of about 10 nm [12]. 
However, they concluded I is proportional to the inverse 
of \M\, distinguished from present discussion. In fact, this 
situation does not belong to NES, but SES, since both 
the decay characteristic quantities Ai,a have image part 
as they stressed, too. Therefore, the penetration depth 
should be independent of The difference between 

NES and SES will be further discussed in the next sec- 
tion. 

Normal And Anomalous Finite Size Effects. 

The finite size effect in QSH system arises from the over- 
lap of the opposite channel due to the decreasing sample 
width, leading to the finite energy gap opening for the en- 
ergy dispersion of edge state near Dirac point [8] . Since the 
penetration depth of NES and SES is quite different, the 
consequent finite size effect is also expected to be distinct. 
We now turn to the ribbon geometry with the bound- 
ary condition of ^(-L/2) = * fcx (L/2) = 0, where L 
is the width of the ribbon. Our numerical results reveal 
that, the relative gap SA(k x ) = A(k x ) — 2| Ak x \ reaches its 
maximum at T point and decays exponentially with \k x \. 
Hence, we just focus on the situation at T point where 
SA(Q) = A(0). We follow the previous discussions [8,14] 
to evaluate the finite size gap in different situations. 

When the NES dominates the Dirac point, as discussed 
previously in the HgTe quantum well [8], the gap was es- 
timated to be 



A(0) 



4|AM 7 | 



-\ 2 L 



sjA 2 - 4MS7 



(6) 



Here we assume \\L >> 1 and Ai >> A2. This is the 
normal finite size effect as shown in Fig. 3(a). When Dirac 
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Fig. 3: Normal and anomalous finite size effects, (a), and (b) 
show the edge bands and gap behaviors varying with L in NES, 
and SES, respectively, (c) gives the M-dependent A(0) with 
the same parameter adopted in Fig. 1 (in logarithmic scale). 
The normal (left), and anomalous (right) finite size effect is di- 
vided by the critical M c (black dash line), (d), and (e) present 
the conductance at finite temperature corresponding to the sit- 
uation of (a), and (b), respectively, with 1/fcsT = 400 and 
Go - -r-. 



point locates inside the SES regime, the gap turns to be 
8|AM 7 sin(3A 2 L)| L 



A(0) 



^A 2 - 4MB j 



(7) 



here QA 2 = J ^ — is the imaginary part of A 2 . The 
gap exhibits an oscillatory behavior with L, as described 
in Fig. 3(b). This oscillation was also predicted in 3D 
topological insulator [12, 14], referred as the anomalous 
finite size effect. 

We numerically investigate the M-dependent evolution 
of A(0) to distinguish the difference between NES and SES 
as shown in Fig. 3(c). Here D = is applied to avoid the 
mismatch between the Dirac point and the regime of SES. 
For small \M\, the Dirac point is NES, and the correspond- 
ing A(0) evolves monotonously with |M|. For large L, the 
Dirac point turns to be SES, A(0) is oscillatory. A critical 
\M C \ = 0.04 is obtained with the same parameters taken 
in Fig. 1. The number of oscillatory periods increase with 
|M|, owing to a decreasing 3 A. Considerable gap always 
opens at L ~ 35 (arb. units) for all \M\ > \M C \, which 
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coincides with the uniform minimal £ses discussed above. 
Here we emphasize that, the uniform minimal £ses found 
in Fig. 1 is protected by the linear dispersion of Eqs. (4). 
This linear relation is not preserved when the finite size 
gap opens, then £ becomes momentum-dependent again 
even in SES. 

The essence of such differences can be understood based 
on present results. For SES, a y-dependent phase factor 
emerges due to the finite 3A, which is absent in NES. The 
edge band is rcnormalizcd, together with gap opening, due 
to the overlap of opposite edge states. Meanwhile, the 
transversal phase coherence of the opposite edge states 
contributes to the oscillation of A for SES. Hence, the 
interference-fringe-like picture can be obtained as shown 
in Fig. 3(c). 

Such effect can be detected in transport measurements 
at low temperature [9] . The conductance at finite temper- 
ature is simply given by [8] 

G(fi, T) = (2e 2 /h) [/(A/2 - /i) — /(-A/2 - /i) + 1] (8) 

when /i locates inside the bulk gap. Here f(E) is the Fermi 
distribution function and A is the finite size gap. Fig. 3(d) 
and Fig. 3(e) present the conductance for NES and SES 
respectively. Recently, a non-monotonous gap evolution 
had been observed in ultra-thin Bi 2 Se 3 , which is noted as 
a possible anomalous finite size effect for TSS [24]. 

Discussion. — Up to now, we have discussed the pen- 
etration depth and the finite size effect in NES and SES. 
The particle-hole asymmetry factor 7 also plays a subtle 
role on these properties. The Dirac point moves upward, 
and the SES regime also shifts, leading to the possible 
mismatch as shown in Fig. 1(e). The existence of SES 
Dirac point requires -gj- < 7^gp- In Fig. 4(b), the regime 
between the solid line and the dash line describes the mis- 
match: the SES exists, but the Dirac point moves outside. 
It should be pointed out that such mismatch is not sensi- 
tive with selected \D/B\ unless it approaches to 1. 

The penetration depth of the edge states at Dirac point 
and finite size effect are discussed together in the phase 
space of relative parameters and as shown in Fig. 4. 
In these phase diagrams, since the solid line divides the pa- 
rameter space into two regimes, £~ l at Dirac point reveals 
two distinct evolutions, £ _1 at Dirac point increases with 
M/B and decreases with |A/B| in NES. In contrast, it re- 
mains unchanged with M/B but increases with |A/i?| in 
SES. The recently discovered topological non-trivial sys- 
tems: 2D HgTe quantum well [8], Ultrathin Bi 2 Se 3 film 
[17], 3D Bi 2 Se 3 , Sb 2 Te 3 and Bi 2 Te 3 [6,25] are compared 
in the same phase space. The former two effective 2D 
systems are described by the same model of Eqs. (1). Al- 
though the 3D topological insulators have a different effec- 
tive model [25], the situation at Dirac point is equivalent 
to the two 2D systems under proper parameter substitu- 
tion [12, 14]. As in Fig. 4(c), the helical edge states in the 
two 2D systems contain only NES, therefore, large size is 
required to avoid the normal finite size effect. In contrast, 
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Fig. 4: Phase diagrams in parameter space, (a), and (b) 
are with, and without the particle-hole symmetry respectively. 
The black solid line (A 2 — 4MB) divides the region of edge 
state into SES and NES. The dash line (A 2 = 4-yMB) in (b) 
indicates the mismatch between SES and the Dirac point as de- 
scribed in the text, and moves along the direction indicated by 
arrow when 7 decreases. The intensity stands for the inverse of 
penetration depth £~ x at Dirac point. Different materials are 
compared in (c), where A, B and M are unified into the units 
of meV and nm, so that £ has a common unit of nm. Here 
the logarithmic scale is adopted. Materials with/without SES 
Dirac point are marked with cycles/squares. The intensity in 
(c) is for 7 = 1. For Bi 2 Se 3 TSS, and Sb 2 Te 3 , \D/B\ is 0.13, 
and 0.63, respectively. 

the 2D TSS of 3D Bi 2 Se 3 and Sb 2 Te 3 implies a shorter 
penetration depth and a possible anomalous finite size ef- 
fect [12,24]. Interestingly, the SES exits in Bi 2 Te 3 , how- 
ever, its Dirac point moves into bulk state due to strong 
particle-hole asymmetry mentioned above. This may be 
true as compared with the angle resolved photoemission 
spectroscopy measurements [6,26]. Similar behavior may 
also can be found in the bulk HgTe under uniaxial strain 
[27]. We expect that, these special effects can be electri- 
cally detected in other QSH systems with smaller A/B or 
larger M/B as shown in Fig. 4. Recently, several designs 
had been performed based on the finite size effect [10,11]. 
The future applications could be quite sensitive to these 
properties. In this sense, present work provides a theo- 
retical prediction on the possible finite size effects in new 
materials. 

Conclusion. — In conclusion, two different transver- 
sal propagation modes of the helical edge states, i.e., NES 
and SES, in the QSH system are specified by the decay 
characteristic quantities A. The emergence of the flat bulk 
band implies the special edge state, which gives a sufficient 
criterion to distinguish the two modes. The penetration 
depth of SES keeps a uniform minimal value, independent 
of the selected E, k x and M. In contrast, it is much larger 
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and shows clear momentum dependence in NES. Different 
finite size effects are studied in respective edge states. Es- 
pecially, the oscillatory gap for edge band is found in SES. 
Some real materials are compared in the phase diagram 
to demonstrate the difference between NES and SES. We 
also give clues to search possible QSH materials with SES 
for future applications. 
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